Computational models of socially interactive animation

EThOS - Electronic Theses Online ServiceGBUnited Kingdo

Computational Models of Socially Interactive Animation.

The aim of this thesis is to investigate computational models of non-verbal social interaction for the purpose of generating synthetic social behaviour in animations. To this end, several contributions are made: A dynamic model, providing multimodal control of animation is developed and demonstrated using various data formats including motion capture and video; A social interaction model is developed, capable of predicting social context/intent such as level of interest in a conversation; and finally, the social model is used to drive the dynamic model, which animates appropriate social behaviour of a listener in a conversation in response to a speaker. A method of reusing motion captured data by learning a generative model of motion is presented. The model allows real-time synthesis and blending of motion, whilst providing it with the style and realism present in the original data set. This is achieved by projecting the data into a lower dimensional space and learning a multivariate probability distribution of the motion sequences. Functioning as a generative model, the probability density estimation is used to produce novel poses, and pre-computed motion derivatives combined with gradient based optimisation generates the animation. A new algorithm for real-time interactive motion control is introduced and demonstrated on motion captured data, pre-recorded videos and HCI. This example-based method uses the original motion data for synthesis by seamlessly combining various subsequences together. A novel approach to determining transition points is presented based on k-medoids, whereby appropriate points of intersection in the motion trajectory are derived as cluster centres. These points are used to segment the data into smaller subsequences. A transition matrix combined with a kernel density estimation is used to determine suitable transitions between the subsequences to develop novel motion. To facilitate real-time interactive control, conditional probabilities are used to derive motion given user commands. The user control can come from any modality including auditory, touch and gesture. The system is also extended to HCI using audio signals from speech in a conversation to trigger non-verbal responses from a synthetic listener in real-time. The flexibility of the method is demonstrated by presenting results ranging from data sets composed of vectorised images, 2D and 3D point representations. In order to learn the dynamics of social interaction, experiments are conducted to elicit natural social dynamics of people in a conversation. Semi-supervised computer vision techniques are then employed to extract social signals such as laughing and nodding. Learning is performed using association rule data mining to deduce frequently occurring patterns of social trends between a speaker and listener in both interested and not interested social scenarios. The confidence values from rules are utilised to build a Social Dynamics Model (SDM), that can then be used for both classification and visualisation. By visualising the rules generated in the SDM, analysing distinct social trends between an interested and not interested listener in a conversation is possible. The confidence values extracted from the mining can also be used as conditional probabilities to animate social responsive avatars. A texture motion graph is combined with the example-based animation system developed earlier within the thesis. Using the mined rules of social interaction, social signals are synthesised within the animation, providing the user with control over who speaks and the interest level of the participants

A Generative Model for Motion Synthesis and Blending Using Probability Density Estimation

The main focus of this paper is to present a method of reusing motion captured data by learning a generative model of motion. The model allows synthesis and blending of cyclic motion, whilst providing it with the style and realism present in the original data. This is achieved by projecting the data into a lower dimensional space and learning a multivariate probability distribution of the motion sequences. Functioning as a generative model, the probability density estimation is used to produce novel motions from the model and gradient based optimisation used to generate the final animation. Results show plausible motion generation and lifelike blends between different actions

Real-Time Motion Control Using Pose Space Probability Density Estimation

The ability to control the movements of an object or person in a video sequence has applications in the movie and animation industries, and in HCI. In this paper, we introduce a new algorithm for real-time motion control and demonstrate its application to pre-recorded video clips and HCI. Firstly, a dataset of video frames are projected into a lower dimension space. A k-mediod clustering algorithm with a distance metric is used to determine groups of similar frames which operate as cut points, segmenting the data into smaller subsequences. A multivariate probability distribution is learnt and probability density estimation is used to determine transitions between the subsequences to develop novel motion. To facilitate real-time control, conditional probabilities are used to derive motion given user commands. The motion controller is extended to HCI using speech Mel-Frequency Ceptral Coefficients (MFCCs) to trigger movement from an input speech signal. We demonstrate the flexibility of the model by presenting results ranging from datasets composed of both vectorised images and 2D point representation. Results show plausible motion generation and lifelike blends between different types of movement. 1

Visualisation and Prediction of Conversation Interest through Mined Social Signals

Abstract — This paper introduces a novel approach to social behaviour recognition governed by the exchange of non-verbal cues between people. We conduct experiments to try and deduce distinct rules that dictate the social dynamics of people in a conversation, and utilise semi-supervised computer vision techniques to extract their social signals such as laughing and nodding. Data mining is used to deduce frequently occurring patterns of social trends between a speaker and listener in both interested and not interested social scenarios. The confidence values from rules are utilised to build a Social Dynamic Model (SDM), that can then be used for classification and visualisation. By visualising the rules generated in the SDM, we can analyse distinct social trends between an interested and not interested listener in a conversation. Results show that these distinctions can be applied generally and used to accurately predict conversational interest. I